Skip to content

ci: Add temporary gitconfig to clone private dependency over SSH #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 12, 2023
Merged

ci: Add temporary gitconfig to clone private dependency over SSH #3

merged 1 commit into from
May 12, 2023

Conversation

simonjbeaumont
Copy link
Collaborator

@simonjbeaumont simonjbeaumont commented May 11, 2023

Motivation

Until the swift-openapi-runtime repository is made public, it cannot be cloned using a HTTPS package URL without authentication, which is what this package is using in its Package.swift.

This means CI isn't able to run for this package. However, CI is able to clone this repository over SSH, which presents an opportunity for a temporary workaround.

Modifications

Add a temporary step to the Docker Compose CI flow, which creates a Git config in a shared ephemeral volume with the following contents:

[url "git@github.com:apple/swift-openapi-runtime"]
        insteadOf = https://github.com/apple/swift-openapi-runtime

Result

The CI should be able to clone the private dependency and then succeed.

Test Plan

Locally, this works:

docker-compose -f docker/docker-compose.yaml -f docker/docker-compose.2204.58.yaml run test
...
+ swift test -Xswiftc -warnings-as-errors --explicit-target-dependency-import-check error
...
Fetching https://github.com/apple/swift-openapi-runtime
...
Fetched https://github.com/apple/swift-openapi-runtime (2.68s)
...
Executed 86 tests, with 0 failures (0 unexpected) in 3.265 (3.265) seconds

Also, if we use can use the shell Docker Compose service to see things are configured correctly:

root@20ab69e3f292:/code# echo $GIT_CONFIG_GLOBAL
/ci-gitconfig/gitconfig
root@20ab69e3f292:/code# cat $GIT_CONFIG_GLOBAL
[url "git@github.com:apple/swift-openapi-runtime"]
        insteadOf = https://github.com/apple/swift-openapi-runtime

@simonjbeaumont
Copy link
Collaborator Author

Hey @yim-lee. I think the soundness pipeline for this repo has been configured differently from the swift-openapi-runtime CI.

In runtime CI we see:

docker-compose -f docker/docker-compose.yaml -p swift-openapi-runtime-soundness-pr1b run --remind me about this soundness

link

But for this repo, we see the following command being run:

docker-compose -f docker/docker-compose.yaml -p swift-openapi-generator-soundness-pr1b run -v ****:/root/.ssh/id_rsa:z --rm shell -c 'mkdir -p -m -0600 /etc/ssh && ssh-keyscan github.com >> /etc/ssh/ssh_known_hosts && swift -version && uname -a && ./scripts/soundness.sh'

link

Maybe this was because you were trying to workaround the private dependency?

In this case, I think this PR solves the issue in a way that allows the pipelines to be configured "normally".

Would you mind reverting the change in CI (if you made one) to confirm?

Just confirmed that both of these commands run fine locally:

  • docker-compose -f docker/docker-compose.yaml -f docker/docker-compose.2204.58.yaml run test
  • docker-compose -f docker/docker-compose.yaml run soundness

@simonjbeaumont simonjbeaumont marked this pull request as ready for review May 11, 2023 10:53
@simonjbeaumont simonjbeaumont requested a review from czechboy0 May 11, 2023 10:54
@simonjbeaumont simonjbeaumont mentioned this pull request May 11, 2023
@yim-lee
Copy link
Member

yim-lee commented May 11, 2023

@swift-server-bot test this please

@yim-lee
Copy link
Member

yim-lee commented May 11, 2023

A run without the CI change: https://ci.swiftserver.group/job/swift-openapi-generator-soundness-pr1b/9/console

CI has to inject SSH key into the docker container for cloning to work and that's what the change does.

@yim-lee
Copy link
Member

yim-lee commented May 11, 2023

@swift-server-bot test this please

@yim-lee
Copy link
Member

yim-lee commented May 11, 2023

Soundness check failed because of this error:

** ERROR: ❌ SWIFTFORMAT_BIN unset and no swift-format on PATH

Probably because we didn't set option -l for bash. I added the missing options in CI config and soundness check passes now. https://ci.swiftserver.group/job/swift-openapi-generator-soundness-pr1b/10/console

@czechboy0
Copy link
Contributor

@yim-lee On the CI user, is the SSH key not placed in ~/.ssh? If it were, it should "just" work, as the docker-compose command mounts the current user's ~/.ssh and should be able to use those keys: https://github.com/apple/swift-openapi-generator/blob/main/docker/docker-compose.yaml#L23

@yim-lee
Copy link
Member

yim-lee commented May 11, 2023

On the CI user, is the SSH key not placed in ~/.ssh?

Unfortunately CI isn't set up quite like that so this doesn't work. When we ran into this issue in the past for a different project, we tried a few things and the change applied here was the one that worked.

@simonjbeaumont
Copy link
Collaborator Author

Thanks for working on this Yim. We'll merge this and roll out a similar change in the transport repos.

@simonjbeaumont simonjbeaumont merged commit 9c3e440 into apple:main May 12, 2023
czechboy0 added a commit to apple/swift-openapi-urlsession that referenced this pull request May 12, 2023
Use an SSH URL to fetch Runtime in CI

(Same as: apple/swift-openapi-generator#3)

### Motivation

Until the swift-openapi-runtime repository is made public, it cannot be
cloned using a HTTPS package URL without authentication, which is what
this package is using in its Package.swift.

This means CI isn't able to run for this package. However, CI is able to
clone this repository over SSH, which presents an opportunity for a
temporary workaround.

### Modifications

Add a temporary step to the Docker Compose CI flow, which creates a Git
config in a shared ephemeral volume with the following contents:

```gitconfig
[url "git@github.com:apple/swift-openapi-runtime"]
        insteadOf = https://github.com/apple/swift-openapi-runtime
```

### Result

The CI should be able to clone the private dependency and then succeed.

### Test Plan

Locally, this works:

```console
❯ docker-compose -f docker/docker-compose.yaml -f docker/docker-compose.2204.58.yaml run test
...
+ swift test -Xswiftc -warnings-as-errors --explicit-target-dependency-import-check error
...
Fetching https://github.com/apple/swift-openapi-runtime
...
Fetched https://github.com/apple/swift-openapi-runtime (2.68s)
...
Executed 86 tests, with 0 failures (0 unexpected) in 3.265 (3.265) seconds
```

Also, if we use can use the `shell` Docker Compose service to see things
are configured correctly:

```console
root@20ab69e3f292:/code# echo $GIT_CONFIG_GLOBAL
/ci-gitconfig/gitconfig
root@20ab69e3f292:/code# cat $GIT_CONFIG_GLOBAL
[url "git@github.com:apple/swift-openapi-runtime"]
        insteadOf = https://github.com/apple/swift-openapi-runtime
```


Reviewed by: simonjbeaumont

Builds:
     ✔︎ pull request validation (5.8) - Build finished. 
     ✔︎ pull request validation (5.9) - Build finished. 
     ✔︎ pull request validation (nightly) - Build finished. 
     ✔︎ pull request validation (soundness) - Build finished. 

#3
czechboy0 added a commit to swift-server/swift-openapi-async-http-client that referenced this pull request May 12, 2023
Use an SSH URL to fetch Runtime in CI

(Same as: apple/swift-openapi-generator#3)

### Motivation

Until the swift-openapi-runtime repository is made public, it cannot be
cloned using a HTTPS package URL without authentication, which is what
this package is using in its Package.swift.

This means CI isn't able to run for this package. However, CI is able to
clone this repository over SSH, which presents an opportunity for a
temporary workaround.

### Modifications

Add a temporary step to the Docker Compose CI flow, which creates a Git
config in a shared ephemeral volume with the following contents:

```gitconfig
[url "git@github.com:apple/swift-openapi-runtime"]
        insteadOf = https://github.com/apple/swift-openapi-runtime
```

### Result

The CI should be able to clone the private dependency and then succeed.

### Test Plan

Locally, this works:

```console
❯ docker-compose -f docker/docker-compose.yaml -f docker/docker-compose.2204.58.yaml run test
...
+ swift test -Xswiftc -warnings-as-errors --explicit-target-dependency-import-check error
...
Fetching https://github.com/apple/swift-openapi-runtime
...
Fetched https://github.com/apple/swift-openapi-runtime (2.68s)
...
Executed 86 tests, with 0 failures (0 unexpected) in 3.265 (3.265) seconds
```

Also, if we use can use the `shell` Docker Compose service to see things
are configured correctly:

```console
root@20ab69e3f292:/code# echo $GIT_CONFIG_GLOBAL
/ci-gitconfig/gitconfig
root@20ab69e3f292:/code# cat $GIT_CONFIG_GLOBAL
[url "git@github.com:apple/swift-openapi-runtime"]
        insteadOf = https://github.com/apple/swift-openapi-runtime
```


Reviewed by: simonjbeaumont

Builds:
     ✔︎ pull request validation (5.8) - Build finished. 
     ✔︎ pull request validation (5.9) - Build finished. 
     ✔︎ pull request validation (nightly) - Build finished. 
     ✔︎ pull request validation (soundness) - Build finished. 

#3
yim-lee added a commit that referenced this pull request May 30, 2023
### Motivation

The repo is public now. There is no need for the workaround anymore.

### Modifications

Remove gitconfig workaround added in
#3
@czechboy0 czechboy0 added the semver/none No version bump required. label Jun 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver/none No version bump required.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants